Long document retrieval aims to fetch query-relevant documents from a large-scale collection, where knowledge distillation has become de facto to improve a retriever by mimicking a heterogeneous yet powerful cross-encoder. However, in contrast to passages or sentences, retrieval on long documents suffers from the scope hypothesis that a long document may cover multiple topics. This maximizes their structure heterogeneity and poses a granular-mismatch issue, leading to an inferior distillation efficacy. In this work, we propose a new learning framework, fine-grained distillation (FGD), for long-document retrievers. While preserving the conventional dense retrieval paradigm, it first produces global-consistent representations crossing different fine granularity and then applies multi-granular aligned distillation merely during training. In experiments, we evaluate our framework on two long-document retrieval benchmarks, which show state-of-the-art performance.

To improve the performance of the dual-encoder retriever, one effective approach is knowledge distillation from the cross-encoder ranker. Existing works construct the candidate passages following the supervised learning setting where a query is paired with a positive passage and a batch of negatives. However, through empirical observation, we find that even the hard negatives from advanced methods are still too trivial for the teacher to distinguish, preventing the teacher from transferring abundant dark knowledge to the student through its soft label. To alleviate this issue, we propose ADAM, a knowledge distillation framework that can better transfer the dark knowledge held in the teacher with Adaptive Dark exAMples. Different from previous works that only rely on one positive and hard negatives as candidate passages, we create dark examples that all have moderate relevance to the query through mixing-up and masking in discrete space. Furthermore, as the quality of knowledge held in different training instances varies as measured by the teacher's confidence score, we propose a self-paced distillation strategy that adaptively concentrates on a subset of high-quality instances to conduct our dark-example-based knowledge distillation to help the student learn better. We conduct experiments on two widely-used benchmarks and verify the effectiveness of our method.

Dense retrieval aims to map queries and passages into low-dimensional vector space for efficient similarity measuring, showing promising effectiveness in various large-scale retrieval tasks. Since most existing methods commonly adopt pre-trained Transformers (e.g. BERT) for parameter initialization, some work focuses on proposing new pre-training tasks for compressing the useful semantic information from passages into dense vectors, achieving remarkable performances. However, it is still challenging to effectively capture the rich semantic information and relations about passages into the dense vectors via one single particular pre-training task. In this work, we propose a multi-task pre-trained model, MASTER, that unifies and integrates multiple pre-training tasks with different learning objectives under the bottlenecked masked autoencoder architecture. Concretely, MASTER utilizes a multi-decoder architecture to integrate three types of pre-training tasks: corrupted passages recovering, related passage recovering and PLMs outputs recovering. By incorporating a shared deep encoder, we construct a representation bottleneck in our architecture, compressing the abundant semantic information across tasks into dense vectors. The first two types of tasks concentrate on capturing the semantic information of passages and relationships among them within the pre-training corpus. The third one can capture the knowledge beyond the corpus from external PLMs (e.g. GPT-2). Extensive experiments on several large-scale passage retrieval datasets have shown that our approach outperforms the previous state-of-the-art dense retrieval methods. Our code and data are publicly released in https://github.com/microsoft/SimXNS

Knowledge distillation is often used to transfer knowledge from a strong teacher model to a relatively weak student model. Traditional knowledge distillation methods include response-based methods and feature-based methods. Response-based methods are used the most widely but suffer from lower upper limit of model performance, while feature-based methods have constraints on the vocabularies and tokenizers. In this paper, we propose a tokenizer-free method liberal feature-based distillation (LEAD). LEAD aligns the distribution between teacher model and student model, which is effective, extendable, portable and has no requirements on vocabularies, tokenizer, or model architecture. Extensive experiments show the effectiveness of LEAD on several widely-used benchmarks, including MS MARCO Passage, TREC Passage 19, TREC Passage 20, MS MARCO Document, TREC Document 19 and TREC Document 20.

This paper presents E5, a family of state-of-the-art text embeddings that transfer well to a wide range of tasks. The model is trained in a contrastive manner with weak supervision signals from our curated large-scale text pair dataset (called CCPairs). E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts such as retrieval, clustering, and classification, achieving strong performance in both zero-shot and fine-tuned settings. We conduct extensive evaluations on 56 datasets from the BEIR and MTEB benchmarks. For zero-shot settings, E5 is the first model that outperforms the strong BM25 baseline on the BEIR retrieval benchmark without using any labeled data. When fine-tuned, E5 obtains the best results on the MTEB benchmark, beating existing embedding models with 40x more parameters.

知识蒸馏是将知识从强大的教师转移到有效的学生模型的有效方法。理想情况下，我们希望老师越好，学生越好。但是，这种期望并不总是成真。通常，由于教师和学生之间的不可忽略的差距，更好的教师模型通过蒸馏导致不良学生。为了弥合差距，我们提出了一种渐进式蒸馏方法，以进行致密检索。产品由教师渐进式蒸馏和数据进行渐进的蒸馏组成，以逐步改善学生。我们对五个广泛使用的基准，MARCO通道，TREC Passage 19，TREC文档19，MARCO文档和自然问题进行了广泛的实验，其中POD在蒸馏方法中实现了密集检索的最新方法。代码和模型将发布。

基于语义空间中密集表示的检索模型已成为第一阶段检索的必不可少的分支。这些检索员受益于代表学习朝着压缩全球序列级嵌入的进步。但是，它们很容易忽略本地的显着短语和实体在文本中提到的，这些短语通常在第一阶段的检索中扮演枢轴角色。为了减轻这种弱点，我们提议使一个密集的检索器对齐一个表现出色的词典意识代表模型。对齐方式是通过弱化的知识蒸馏来实现的，以通过两个方面来启发猎犬 - 1）词汇扬声的对比目标，以挑战密集编码器和2）一个配对的等级正规化，以使密集的模型的行为倾向于其他人的行为。我们在三个公共基准上评估了我们的模型，这表明，凭借可比的词典觉得回收犬作为老师，我们提议的密集人可以带来一致而重大的改进，甚至超过教师。此外，我们发现我们对密集猎犬的改进是与标准排名蒸馏的补充，这可以进一步提高最先进的性能。

Gazetteer广泛用于中文命名实体识别（NER），以增强跨度边界检测和类型分类。然而，为了进一步了解地被拼写器的普遍性和有效性，NLP社区仍然缺乏对宪报增强的NER模型的系统分析。在本文中，我们首先重新审查了宪报增强NER模型的几种常见实践，并进行了一系列详细的分析，以评估模型性能与宪报特征之间的关系，这可以指导我们建立更多合适的宪报。本文的发现如下：（1）宪报改善了最多的情况，即对于传统的NER模型来说，数据集很难学习。（2）模型的性能极大地受益于高质量的预训练的词汇嵌入。（3）一个好的地名词典应涵盖更多在训练集和测试集中可以匹配的实体。

本文着重于几次NLP任务的文本数据增强。现有的数据增强算法要么使用一个小型培训集来生成新的合成数据，要么利用与任务无关的启发式规则（例如，同义词替代）或微调通用预训练的语言模型（例如GPT2）。因此，这些方法具有特定于任务的知识，并且仅限于在简单任务中为弱基线产生低质量的合成数据。为了解决这个问题，我们提出了知识混合数据增强模型（KNOWDA）：使用知识混合培训（KOMT）在不同的NLP任务的混合物上预测的编码器LM。 KOMT是一种培训程序，将各种异质NLP任务的输入示例重新定义为统一的文本到文本格式，并采用不同粒度的目标，以学习生成部分或完整的样本。在KOMT的帮助下，Knowda可以隐含地将所需的特定于任务的知识从任务的混合中隐含地结合在一起，并通过一些给定的实例迅速掌握目标任务的固有综合定律。据我们所知，我们是首次尝试将任务数量扩展到多任务共同培训以进行数据扩展。广泛的实验表明，i）Knowda成功地通过少量基准的基准成功地提高了Albert和Deberta的表现，表现优于先前的最新数据增强基线； ii）KNOWDA还可以改善少数弹药任务的模型性能，这是KOMT中未包含的固定任务类型。

可区分的搜索索引（DSI）是一个新的新兴范式，用于信息检索。与索引和检索是两个不同且独立的组件的传统检索体系结构不同，DSI使用单个变压器模型执行索引和检索。在本文中，我们确定并解决了当前DSI模型的重要问题：DSI索引和检索过程之间发生的数据分布不匹配。具体而言，我们认为，在索引时，当前的DSI方法学会学会在长文档文本及其标识之间建立连接，但是在检索中，向DSI模型提供了简短的查询文本以执行文档标识符的检索。当使用DSI进行跨语言检索时，此问题进一步加剧，其中文档文本和查询文本使用不同的语言。为了解决当前DSI模型的这个基本问题，我们为DSI称为DSI-QG的简单而有效的索引框架。在DSI-QG中，文档由索引时间的查询生成模型生成的许多相关查询表示。这允许DSI模型在索引时将文档标识符连接到一组查询文本，因此减轻索引和检索阶段之间存在的数据分布不匹配。流行的单语言和跨语性通过基准数据集的经验结果表明，DSI-QG明显优于原始DSI模型。